Efficient Gradient Approximation Method for Constrained Bilevel Optimization

Bilevel optimization has been developed for many machine learning tasks with large-scale and high-dimensional data. This paper considers a constrained bilevel problem, where the lower-level problem is convex equality inequality constraints upper-level non-convex. The overall objective function non-convex non-differentiable. To solve we develop gradient-based approach, called gradient approximation method, which determines descent direction by computing several representative gradients of inside neighborhood current estimate. We show that algorithm asymptotically converges to set Clarke stationary points, demonstrate efficacy experiments on hyperparameter meta-learning.